Skip to content

Upstream tracking#165

Draft
grahamc wants to merge 3101 commits into
2.34-maintenancefrom
main
Draft

Upstream tracking#165
grahamc wants to merge 3101 commits into
2.34-maintenancefrom
main

Conversation

@grahamc

@grahamc grahamc commented Jul 31, 2025

Copy link
Copy Markdown
Member

Motivation

Not intended to be merged directly. This PR is a convenience to show the diff between upstream Nix and Determinate Nix (the main branch).

Continuation of #4.

@grahamc grahamc requested a review from edolstra as a code owner July 31, 2025 17:14
@github-actions github-actions Bot temporarily deployed to production July 31, 2025 17:14 Inactive
@DeterminateSystems DeterminateSystems locked as off-topic and limited conversation to collaborators Jul 31, 2025
@github-actions github-actions Bot temporarily deployed to pull request July 31, 2025 18:20 Inactive
@github-actions github-actions Bot temporarily deployed to production July 31, 2025 18:21 Inactive
@cole-h cole-h marked this pull request as draft August 1, 2025 14:26
@github-actions github-actions Bot temporarily deployed to pull request August 4, 2025 22:15 Inactive
@github-actions github-actions Bot temporarily deployed to commit August 4, 2025 22:15 Inactive
@github-actions github-actions Bot temporarily deployed to production August 4, 2025 22:15 Inactive
@github-actions github-actions Bot temporarily deployed to production August 5, 2025 14:25 Inactive
@github-actions github-actions Bot temporarily deployed to pull request August 5, 2025 14:25 Inactive
@github-actions github-actions Bot temporarily deployed to pull request August 7, 2025 15:58 Inactive
@github-actions github-actions Bot temporarily deployed to production August 7, 2025 15:58 Inactive
@github-actions github-actions Bot temporarily deployed to pull request August 7, 2025 23:01 Inactive
@github-actions github-actions Bot temporarily deployed to production August 7, 2025 23:02 Inactive
@github-actions github-actions Bot temporarily deployed to pull request August 10, 2025 16:36 Inactive
@github-actions github-actions Bot temporarily deployed to production August 10, 2025 16:36 Inactive
@github-actions github-actions Bot temporarily deployed to pull request August 10, 2025 20:06 Inactive
@github-actions github-actions Bot temporarily deployed to production August 10, 2025 20:06 Inactive
@github-actions github-actions Bot temporarily deployed to production August 19, 2025 15:04 Inactive
@github-actions github-actions Bot temporarily deployed to pull request August 19, 2025 15:04 Inactive
@github-actions github-actions Bot temporarily deployed to production August 20, 2025 10:41 Inactive
@github-actions github-actions Bot temporarily deployed to pull request August 20, 2025 10:41 Inactive
@github-actions github-actions Bot temporarily deployed to commit August 20, 2025 10:41 Inactive
@github-actions github-actions Bot temporarily deployed to pull request August 25, 2025 16:07 Inactive
@github-actions github-actions Bot temporarily deployed to production August 25, 2025 16:07 Inactive
@github-actions github-actions Bot temporarily deployed to production August 25, 2025 16:14 Inactive
cole-h and others added 30 commits June 26, 2026 21:06
…tenance

[Backport 2.34-maintenance] installer: Bail out early if running on macOS < 14.0
Since curl/curl@2a2104f
libcurl now swallows events in curl_multi_perform, so there's a chance that we miss
a wakeup.

This is somewhat reproducible on my machine by doing nix flake prefetch https://channels.nixos.org/nixos-25.11/nixexprs.tar.xz.

Another option is to bring back our own wakeup pipe, but that's less portable.
Ideally this regression would be fixed in libcurl too...

(cherry picked from commit 0d0c333)
…tenance

[Backport 2.34-maintenance] libstore: Fix libcurl thread wakeup with curl >= 8.21
Nixpkgs now uses structured attrs (i.e. env.<VAR>), so this override
wasn't working anymore. It makes a huge performance impact once GC
kicks in (e.g. it speeds up a 12-core `nix search nixpkgs` from 10.5
to 6.5 seconds).

Assisted-by: Claude Fable 5 <noreply@anthropic.com>
…k-size

Fix INITIAL_MARK_STACK_SIZE override in boehm-gc
nix search: Print total number of derivations
The previous implementation used a std::map with traceable_allocator,
which costs one GC_MALLOC_UNCOLLECTABLE() and one GC_FREE() per
distinct attribute name. Uncollectable allocations always take the
global GC allocation lock, making this map a major source of mutex
contention during parallel evaluation (~12% of all futex calls in a
16-core 'nix search' run, plus the corresponding GC_free traffic).

Instead, collect the (name, value) pairs into a plain std::vector and
group them by stable-sorting on the name, which yields the same
attribute order and per-attribute value order as the map. The vector
doesn't need to be visible to the GC since the values it points to
are kept alive by the attrsets in args[1] for the duration of the
call.

At 16 eval cores (with GC disabled via GC_INITIAL_HEAP_SIZE=40G),
this cuts kernel time from ~21s to ~8-9s, context switches from 2.6M
to 1.5M, and elapsed time from ~6.2s to ~4.9s for 'nix search nixpkgs
--no-eval-cache fizzbuzz'. Single-core performance is unchanged.

Assisted-by: Claude Fable 5 <noreply@anthropic.com>
Make zipAttrsWith more parallel-eval friendly
Apply the same optimization to builtins.groupBy as to
builtins.zipAttrsWith (162a34378c): instead of accumulating the groups
in a std::map<Symbol, ValueVector> with traceable_allocator - which
costs a GC_MALLOC_UNCOLLECTABLE() per distinct name plus traceable
growth allocations for every per-group vector, all taking the global
GC allocation lock - collect the (name, value) pairs into a plain
std::vector and group them by stable-sorting on the name. The vector
doesn't need to be visible to the GC since the values it points to
are kept alive by the list in args[1] for the duration of the call.

The shared machinery (the NameValue type, sorting/counting the names,
and iterating over the per-name runs) is factored out into helpers
used by both primops.

This removes the last user of ValueVectorMap, so drop that typedef.

Assisted-by: Claude Fable 5 <noreply@anthropic.com>
builtins.groupBy: Drop ValueVectorMap
Evaluating an empty attrset literal wrote its position into
Bindings::emptyBindings, the shared static object that
EvalMemory::allocBindings() returns for zero-capacity bindings. Under
parallel evaluation this is a data race (the position of every '{ }'
was whichever one was evaluated last), and 'perf c2c' on an Intel
i7-1260P showed it also causes false sharing: emptyBindings happens
to share a cache line with Counter::enabled, which is read by every
thread in allocValue()/callFunction()/maybeThunk(), so each write
invalidated that line across all cores.

Skip the write for the shared empty bindings; '{ }' now has a
deterministic (undefined) position instead of a racy one, and the
cache line stays clean. Verified with perf c2c that the line no
longer appears among contended cache lines.

Assisted-by: Claude Fable 5 <noreply@anthropic.com>
ExprAttrs::eval: Don't write to the shared empty Bindings
Introduce LocalStore::addTempRoots, which adds all temproots in a batch
for LocalStore.

Make this available from the base class Store by making a virtual method
Store::addTempRoots, overriding it in LocalStore and RemoteStore (it is
just a loop for each path because the benefit of having this be batched
is unclear), and making the single-path Store::addTempRoot a wrapper
around that.
Since 734a205, copyPaths() calls
addTempRoot() on the remote store for every path in the
closure. However, this is extremely show for large closures,
especially over high-latency SSH connections. So now, we use a new
daemon operation AddTempRoots to do all paths in a single call.

Fixes #533.

Assisted-by: Claude Fable 5 <noreply@anthropic.com>
Use the new batched addTempRoots for both convenience and better
performance for LocalStore.
Use the new batched addTempRoots for both convenience and better
performance for LocalStore.
Fix very slow remote temp root registration in copyPaths()
This is intended to allow a store to query multiple paths in a single
call.

Taken from #523.

Assisted-by: Claude Fable 5 <noreply@anthropic.com>
Add a batched QueryPathInfos operation to the worker protocol, gated
behind a new "queryPathInfos" protocol feature, and use it to
implement an efficient RemoteStore::queryPathInfos() override.

Previously, querying path infos from a remote store required a
network round-trip per store path, which is extremely slow over
high-latency links (e.g. in computeFSClosure()). Now the entire set
of paths is sent in a single operation. The response contains the
infos for the valid paths; paths not reported are invalid.

The client answers what it can from the in-memory path info cache
first, updates the cache with the results (including negative
entries), and falls back to the per-path base implementation for
daemons that don't support the new operation.

Assisted-by: Claude Fable 5 <noreply@anthropic.com>
Make computeFSClosure() much faster over SSH
Tagging release 2.34.8
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Projects

None yet

Development

Successfully merging this pull request may close these issues.